{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "89cf2628",
   "metadata": {},
   "source": [
    "# State Estimation Example\n",
    "\n",
    "In this notebook we will walk through an example of state estimation calculation of `power-grid-model`.\n",
    "The following points are covered\n",
    " * Construction of the model\n",
    " * Run state estimation once, and its relevant function arguments\n",
    "\n",
    "\n",
    "It serves an example of how to use the Python API. For detailed API documentation, refer to \n",
    "[Python API reference](../api_reference/python-api-reference.md)\n",
    "and [Native Data Interface](../advanced_documentation/native-data-interface.md).\n",
    "\n",
    "\n",
    "## Example Network\n",
    "\n",
    "We use a simple network with 3 nodes, 1 source, 3 lines, and 2 loads. As shown below:\n",
    "\n",
    "```\n",
    " -----------------------line_8---------------\n",
    " |                                          |\n",
    "node_1 ---line_3--- node_2 ----line_5---- node_6\n",
    " |                    |                     |\n",
    "source_10          sym_load_4           sym_load_7\n",
    "```\n",
    "\n",
    "The 3 nodes are connected in a triangular way by 3 lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ae11dc9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# some basic imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from power_grid_model import LoadGenType, DatasetType, ComponentType\n",
    "from power_grid_model import PowerGridModel, CalculationMethod, CalculationType, MeasuredTerminalType\n",
    "from power_grid_model import initialize_array"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f983cef7",
   "metadata": {},
   "source": [
    "## Input Dataset\n",
    "\n",
    "We create input dataset by using the helper function `initialize_array`.\n",
    "Note the units of all input are standard SI unit without any prefix,\n",
    "i.e. the unit of voltage is volt (V), not kV.\n",
    "\n",
    "Please refer [Components](../user_manual/components.md) for detailed explanation of all component types and their input/output attributes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6f008736",
   "metadata": {},
   "outputs": [],
   "source": [
    "# node\n",
    "node = initialize_array(DatasetType.input, ComponentType.node, 3)\n",
    "node[\"id\"] = [1, 2, 6]\n",
    "node[\"u_rated\"] = [10.5e3, 10.5e3, 10.5e3]\n",
    "\n",
    "# line\n",
    "line = initialize_array(DatasetType.input, ComponentType.line, 3)\n",
    "line[\"id\"] = [3, 5, 8]\n",
    "line[\"from_node\"] = [1, 2, 1]\n",
    "line[\"to_node\"] = [2, 6, 6]\n",
    "line[\"from_status\"] = [1, 1, 1]\n",
    "line[\"to_status\"] = [1, 1, 1]\n",
    "line[\"r1\"] = [0.25, 0.25, 0.25]\n",
    "line[\"x1\"] = [0.2, 0.2, 0.2]\n",
    "line[\"c1\"] = [10e-6, 10e-6, 10e-6]\n",
    "line[\"tan1\"] = [0.0, 0.0, 0.0]\n",
    "line[\"i_n\"] = [1000, 1000, 1000]\n",
    "\n",
    "# load\n",
    "sym_load = initialize_array(DatasetType.input, ComponentType.sym_load, 2)\n",
    "sym_load[\"id\"] = [4, 7]\n",
    "sym_load[\"node\"] = [2, 6]\n",
    "sym_load[\"status\"] = [1, 1]\n",
    "sym_load[\"type\"] = [LoadGenType.const_power, LoadGenType.const_power]\n",
    "sym_load[\"p_specified\"] = [20e6, 10e6]\n",
    "sym_load[\"q_specified\"] = [5e6, 2e6]\n",
    "\n",
    "# source\n",
    "source = initialize_array(DatasetType.input, ComponentType.source, 1)\n",
    "source[\"id\"] = [10]\n",
    "source[\"node\"] = [1]\n",
    "source[\"status\"] = [1]\n",
    "source[\"u_ref\"] = [1.0]\n",
    "\n",
    "# voltage sensor\n",
    "sym_voltage_sensor = initialize_array(DatasetType.input, ComponentType.sym_voltage_sensor, 3)\n",
    "sym_voltage_sensor[\"id\"] = [11, 12, 13]\n",
    "sym_voltage_sensor[\"measured_object\"] = [1, 2, 6]\n",
    "sym_voltage_sensor[\"u_sigma\"] = [1.0, 1.0, 1.0]\n",
    "sym_voltage_sensor[\"u_measured\"] = [10489.37, 9997.32, 10102.01]\n",
    "\n",
    "# power sensor\n",
    "sym_power_sensor = initialize_array(DatasetType.input, ComponentType.sym_power_sensor, 8)\n",
    "sym_power_sensor[\"id\"] = [14, 15, 16, 17, 18, 19, 20, 21]\n",
    "sym_power_sensor[\"measured_object\"] = [3, 3, 5, 5, 8, 8, 4, 6]\n",
    "sym_power_sensor[\"measured_terminal_type\"] = [\n",
    "    MeasuredTerminalType.branch_from,\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.branch_from,\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.branch_from,\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.load,\n",
    "    MeasuredTerminalType.node,\n",
    "]\n",
    "sym_power_sensor[\"power_sigma\"] = [1.0e3, 1.0e3, 1.0e3, 1.0e3, 1.0e3, 1.0e3, 1.0e3, 1.0e3]\n",
    "sym_power_sensor[\"p_measured\"] = [1.73e07, -1.66e07, -3.36e06, 3.39e06, 1.38e07, -1.33e07, 20e6, -10e6]\n",
    "sym_power_sensor[\"q_measured\"] = [4.07e06, -3.82e06, -1.17e06, 8.86e05, 2.91e06, -2.88e06, 5e6, -2e6]\n",
    "\n",
    "# all\n",
    "input_data = {\n",
    "    ComponentType.node: node,\n",
    "    ComponentType.line: line,\n",
    "    ComponentType.sym_load: sym_load,\n",
    "    ComponentType.source: source,\n",
    "    ComponentType.sym_voltage_sensor: sym_voltage_sensor,\n",
    "    ComponentType.sym_power_sensor: sym_power_sensor,\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "157db2f0",
   "metadata": {},
   "source": [
    "## Validation (optional)\n",
    "For efficiency reasons, most of the data is not explicitly validated in the power grid model. However, in most cases, a state estimation will fail/crash if the data is invalid. Often with a low level error message that is hard to relate to the original objects. Therfore, it is recommended to always validate your data before constructing a PowerGridModel instance.\n",
    "\n",
    "The simplest and most effective way to validate your data is by using `assert_valid_input_data()` which will throw an error if it encounters any invalid data. See [Validation Examples](./Validation%20Examples.ipynb) for more detailed information on the validation functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fab31f58",
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model.validation import assert_valid_input_data\n",
    "\n",
    "assert_valid_input_data(input_data=input_data, calculation_type=CalculationType.state_estimation)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "afae0224",
   "metadata": {},
   "source": [
    "## Construction\n",
    "\n",
    "The construction of the model is just calling the constructor of `PowerGridModel`. The default `system_frequency` is 50.0 Hz.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7ef134e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = PowerGridModel(input_data)\n",
    "# model = PowerGridModel(input_data, system_frequency=60.0)  # if you have another system frequency"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e3605c3e",
   "metadata": {},
   "source": [
    "## One-time State Estimation Calculation\n",
    "\n",
    "You can call the method `calculate_state_estimation` to do a one-time state estimation calculation based on the current network data in the model. In this case you should not specify the argument `update_data` as it is used for batch calculation.\n",
    "\n",
    "The following command executes a one-time state estimation calculation with (they are also the default values for those arguments)\n",
    "* Symmetric calculation\n",
    "* Iterative linear method\n",
    "* Error tolerance: 1e-8\n",
    "* Maximum number of iteration: 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "44c2de63",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_data = model.calculate_state_estimation(\n",
    "    symmetric=True, error_tolerance=1e-8, max_iterations=20, calculation_method=CalculationMethod.iterative_linear\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d08aaf45",
   "metadata": {},
   "source": [
    "### Result Dataset\n",
    "\n",
    "We can also print one result dataset of node and line by converting the array to dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a581a36e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node result\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.998922</td>\n",
       "      <td>10488.678869</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.112866e+07</td>\n",
       "      <td>6.984619e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.952157</td>\n",
       "      <td>9997.648144</td>\n",
       "      <td>-0.022921</td>\n",
       "      <td>-1.996348e+07</td>\n",
       "      <td>-4.998968e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0.962132</td>\n",
       "      <td>10102.384509</td>\n",
       "      <td>-0.018780</td>\n",
       "      <td>-9.956656e+06</td>\n",
       "      <td>-1.999083e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu             u   u_angle             p             q\n",
       "0   1          1  0.998922  10488.678869  0.000000  3.112866e+07  6.984619e+06\n",
       "1   2          1  0.952157   9997.648144 -0.022921 -1.996348e+07 -4.998968e+06\n",
       "2   6          1  0.962132  10102.384509 -0.018780 -9.956656e+06 -1.999083e+06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Line result\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>loading</th>\n",
       "      <th>p_from</th>\n",
       "      <th>q_from</th>\n",
       "      <th>i_from</th>\n",
       "      <th>s_from</th>\n",
       "      <th>p_to</th>\n",
       "      <th>q_to</th>\n",
       "      <th>i_to</th>\n",
       "      <th>s_to</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.983446</td>\n",
       "      <td>1.731816e+07</td>\n",
       "      <td>4.068528e+06</td>\n",
       "      <td>979.232855</td>\n",
       "      <td>1.778965e+07</td>\n",
       "      <td>-1.659573e+07</td>\n",
       "      <td>-3.820392e+06</td>\n",
       "      <td>983.446244</td>\n",
       "      <td>1.702978e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.206048</td>\n",
       "      <td>-3.367751e+06</td>\n",
       "      <td>-1.178575e+06</td>\n",
       "      <td>206.048359</td>\n",
       "      <td>3.568023e+06</td>\n",
       "      <td>3.398729e+06</td>\n",
       "      <td>8.860393e+05</td>\n",
       "      <td>200.729011</td>\n",
       "      <td>3.512325e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>0.780866</td>\n",
       "      <td>1.381050e+07</td>\n",
       "      <td>2.916091e+06</td>\n",
       "      <td>776.961713</td>\n",
       "      <td>1.411500e+07</td>\n",
       "      <td>-1.335538e+07</td>\n",
       "      <td>-2.885123e+06</td>\n",
       "      <td>780.865585</td>\n",
       "      <td>1.366346e+07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized   loading        p_from        q_from      i_from  \\\n",
       "0   3          1  0.983446  1.731816e+07  4.068528e+06  979.232855   \n",
       "1   5          1  0.206048 -3.367751e+06 -1.178575e+06  206.048359   \n",
       "2   8          1  0.780866  1.381050e+07  2.916091e+06  776.961713   \n",
       "\n",
       "         s_from          p_to          q_to        i_to          s_to  \n",
       "0  1.778965e+07 -1.659573e+07 -3.820392e+06  983.446244  1.702978e+07  \n",
       "1  3.568023e+06  3.398729e+06  8.860393e+05  200.729011  3.512325e+06  \n",
       "2  1.411500e+07 -1.335538e+07 -2.885123e+06  780.865585  1.366346e+07  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sym_load result\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "      <th>i</th>\n",
       "      <th>s</th>\n",
       "      <th>pf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1.996348e+07</td>\n",
       "      <td>4.998968e+06</td>\n",
       "      <td>1188.457478</td>\n",
       "      <td>2.057985e+07</td>\n",
       "      <td>0.970050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>9.956656e+06</td>\n",
       "      <td>1.999083e+06</td>\n",
       "      <td>580.377789</td>\n",
       "      <td>1.015536e+07</td>\n",
       "      <td>0.980434</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized             p             q            i             s  \\\n",
       "0   4          1  1.996348e+07  4.998968e+06  1188.457478  2.057985e+07   \n",
       "1   7          1  9.956656e+06  1.999083e+06   580.377789  1.015536e+07   \n",
       "\n",
       "         pf  \n",
       "0  0.970050  \n",
       "1  0.980434  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Node result\")\n",
    "display(pd.DataFrame(output_data[ComponentType.node]))\n",
    "\n",
    "print(\"Line result\")\n",
    "display(pd.DataFrame(output_data[ComponentType.line]))\n",
    "\n",
    "print(\"Sym_load result\")\n",
    "display(pd.DataFrame(output_data[ComponentType.sym_load]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2c7c914",
   "metadata": {},
   "source": [
    "### Calculation Method\n",
    "\n",
    "There are two calculation methods available. They are listed in the `CalculationMethod` enumeration as follows:\n",
    "* **Iterative linear method**: \n",
    "    * `CalculationMethod.iterative_linear`: All measurements are linearized before running the method.\n",
    "* **Newton Raphson method**: \n",
    "    * `CalculationMethod.newton_raphson`: traditional Newton-Raphson method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00be7ae5",
   "metadata": {},
   "source": [
    "**Following command does state estimation with Newton Raphson method. Note the difference of result for node compared to the results from iterative linear method.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "40e6d425",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node result\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999187</td>\n",
       "      <td>10491.465789</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.112736e+07</td>\n",
       "      <td>6.981011e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.952440</td>\n",
       "      <td>10000.623189</td>\n",
       "      <td>-0.022911</td>\n",
       "      <td>-1.996275e+07</td>\n",
       "      <td>-4.997141e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0.962411</td>\n",
       "      <td>10105.311164</td>\n",
       "      <td>-0.018773</td>\n",
       "      <td>-9.956895e+06</td>\n",
       "      <td>-1.998502e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu             u   u_angle             p             q\n",
       "0   1          1  0.999187  10491.465789  0.000000  3.112736e+07  6.981011e+06\n",
       "1   2          1  0.952440  10000.623189 -0.022911 -1.996275e+07 -4.997141e+06\n",
       "2   6          1  0.962411  10105.311164 -0.018773 -9.956895e+06 -1.998502e+06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "output_data_NR = model.calculate_state_estimation(symmetric=True, calculation_method=CalculationMethod.newton_raphson)\n",
    "\n",
    "print(\"Node result\")\n",
    "display(pd.DataFrame(output_data_NR[ComponentType.node]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a90ab8fb",
   "metadata": {},
   "source": [
    "It is also possible to specify the standard deviations of active and reactive power mesurements seperately. This is especially powerful when using the Newton Raphson method for state estimation. Refer to the [Documentation](../user_manual/calculations.md) for a detailed explanation. Below is an example of state estimation with seperate standard deviations for p and q measurements.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "699e31af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node result\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.998937</td>\n",
       "      <td>10488.835308</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.110692e+07</td>\n",
       "      <td>6.979313e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.952221</td>\n",
       "      <td>9998.324909</td>\n",
       "      <td>-0.022891</td>\n",
       "      <td>-1.993051e+07</td>\n",
       "      <td>-4.995857e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0.962157</td>\n",
       "      <td>10102.645852</td>\n",
       "      <td>-0.018779</td>\n",
       "      <td>-9.969948e+06</td>\n",
       "      <td>-1.998607e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu             u   u_angle             p             q\n",
       "0   1          1  0.998937  10488.835308  0.000000  3.110692e+07  6.979313e+06\n",
       "1   2          1  0.952221   9998.324909 -0.022891 -1.993051e+07 -4.995857e+06\n",
       "2   6          1  0.962157  10102.645852 -0.018779 -9.969948e+06 -1.998607e+06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# power sensor\n",
    "sym_power_sensor2 = initialize_array(DatasetType.input, ComponentType.sym_power_sensor, 8)\n",
    "sym_power_sensor2[\"id\"] = [14, 15, 16, 17, 18, 19, 20, 21]\n",
    "sym_power_sensor2[\"measured_object\"] = [3, 3, 5, 5, 8, 8, 4, 6]\n",
    "sym_power_sensor2[\"measured_terminal_type\"] = [\n",
    "    MeasuredTerminalType.branch_from,\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.branch_from,\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.branch_from,\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.load,\n",
    "    MeasuredTerminalType.node,\n",
    "]\n",
    "sym_power_sensor2[\"p_measured\"] = [1.73e07, -1.66e07, -3.36e06, 3.39e06, 1.38e07, -1.33e07, 20e6, -10e6]\n",
    "sym_power_sensor2[\"q_measured\"] = [4.07e06, -3.82e06, -1.17e06, 8.86e05, 2.91e06, -2.88e06, 5e6, -2e6]\n",
    "sym_power_sensor2[\"p_sigma\"] = [\n",
    "    1.0e3,\n",
    "    1.0e3,\n",
    "    1.0e4,\n",
    "    1.0e3,\n",
    "    1.0e4,\n",
    "    1.0e3,\n",
    "    1.0e4,\n",
    "    1.0e3,\n",
    "]  # define different standard deviations for p and q measurements\n",
    "sym_power_sensor2[\"q_sigma\"] = [1.0e4, 1.0e4, 1.0e3, 1.0e4, 1.0e3, 1.0e4, 1.0e3, 1.0e4]\n",
    "\n",
    "# all\n",
    "input_data2 = {\n",
    "    ComponentType.node: node,\n",
    "    ComponentType.line: line,\n",
    "    ComponentType.sym_load: sym_load,\n",
    "    ComponentType.source: source,\n",
    "    ComponentType.sym_voltage_sensor: sym_voltage_sensor,\n",
    "    ComponentType.sym_power_sensor: sym_power_sensor2,\n",
    "}\n",
    "\n",
    "model2 = PowerGridModel(input_data2)\n",
    "output_data_NR2 = model2.calculate_state_estimation(symmetric=True, calculation_method=CalculationMethod.newton_raphson)\n",
    "print(\"Node result\")\n",
    "display(pd.DataFrame(output_data_NR2[ComponentType.node]))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e516b436",
   "metadata": {},
   "source": [
    "## Observability\n",
    "\n",
    "In order to perform a state estimation the number of measurements should be larger than or equal to the number of unknowns. For each node there are two unknowns: `u` and `u_angle`.\n",
    "\n",
    "$$n_{\\text{measurements}} >= n_{\\text{unknowns}}$$\n",
    "\n",
    "Where\n",
    "\n",
    "$$n_{\\text{unknowns}} = 2 n_{\\text{nodes}} $$\n",
    "\n",
    "And\n",
    "\n",
    "$$n_{\\text{measurements}} = n_{\\text{nodes\\_with\\_voltage\\_sensor\\_without\\_angle}} + 2 n_{\\text{nodes\\_with\\_voltage\\_sensor\\_with\\_angle}} + 2 n_{\\text{branches\\_with\\_power\\_sensor}} + 2 n_{\\text{nodes\\_without\\_any\\_appliances\\_connected}} + 2 n_{\\text{nodes\\_with\\_all\\_connected\\_appliances\\_measured\\_by\\_power\\_sensor}}$$\n",
    "\n",
    "In this formula, only measurements with non-negligible contributions should be considered. That means that measurements with infinite variances have vanishing contribution and are to be omitted, regardless of whether those divergences are due to infinite sigmas or unreasonably large ones. As a result, missing data can be representing by providing infinity as a standard deviation, and the underdetermined edge case is handled correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc598467",
   "metadata": {},
   "source": [
    "### Some missing measurements\n",
    "\n",
    "Few missing measurements may keep the data observable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b3e3fdc9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.998927</td>\n",
       "      <td>10488.737113</td>\n",
       "      <td>3.391770e-21</td>\n",
       "      <td>3.110041e+07</td>\n",
       "      <td>6.980842e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.952178</td>\n",
       "      <td>9997.866792</td>\n",
       "      <td>-2.291293e-02</td>\n",
       "      <td>-1.997486e+07</td>\n",
       "      <td>-4.998422e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0.962193</td>\n",
       "      <td>10103.023556</td>\n",
       "      <td>-1.874179e-02</td>\n",
       "      <td>-9.918677e+06</td>\n",
       "      <td>-1.997233e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu             u       u_angle             p  \\\n",
       "0   1          1  0.998927  10488.737113  3.391770e-21  3.110041e+07   \n",
       "1   2          1  0.952178   9997.866792 -2.291293e-02 -1.997486e+07   \n",
       "2   6          1  0.962193  10103.023556 -1.874179e-02 -9.918677e+06   \n",
       "\n",
       "              q  \n",
       "0  6.980842e+06  \n",
       "1 -4.998422e+06  \n",
       "2 -1.997233e+06  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sym_voltage_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_voltage_sensor, 1)\n",
    "# for each mutation, only one object is specified\n",
    "sym_voltage_sensor_update[\"id\"] = 13\n",
    "sym_voltage_sensor_update[\"u_sigma\"] = np.inf  # disable this sensor\n",
    "\n",
    "sym_power_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_power_sensor, 1)\n",
    "sym_power_sensor_update[\"id\"] = 21\n",
    "sym_power_sensor_update[\"power_sigma\"] = np.inf  # disable this sensor\n",
    "\n",
    "update_data = {\n",
    "    ComponentType.sym_voltage_sensor: sym_voltage_sensor_update,\n",
    "    ComponentType.sym_power_sensor: sym_power_sensor_update,\n",
    "}\n",
    "\n",
    "# Permanent update\n",
    "model.update(update_data=update_data)\n",
    "output_data = model.calculate_state_estimation()\n",
    "display(pd.DataFrame(output_data[ComponentType.node]))\n",
    "\n",
    "# fully reinstigate the original model to prevent this cell from influencing re-runs. A model update would be sufficient as well.\n",
    "model = PowerGridModel(input_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd22b716",
   "metadata": {},
   "source": [
    "### Unobservable case\n",
    "\n",
    "If too many measurements are invalid, the result may become unobservable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "445726f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not enough measurements available for state estimation.\n",
      "\n",
      "Try validate_input_data() or validate_batch_data() to validate your data.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from power_grid_model.errors import PowerGridError\n",
    "\n",
    "sym_voltage_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_voltage_sensor, 3)\n",
    "sym_voltage_sensor_update[\"id\"] = sym_voltage_sensor[\"id\"]\n",
    "sym_voltage_sensor_update[\"u_sigma\"] = np.inf  # disable all sensors\n",
    "\n",
    "sym_power_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_power_sensor, 8)\n",
    "sym_power_sensor_update[\"id\"] = sym_power_sensor[\"id\"]\n",
    "sym_power_sensor_update[\"power_sigma\"] = np.inf  # disable all sensors\n",
    "\n",
    "update_data = {\n",
    "    ComponentType.sym_voltage_sensor: sym_voltage_sensor_update,\n",
    "    ComponentType.sym_power_sensor: sym_power_sensor_update,\n",
    "}\n",
    "\n",
    "try:\n",
    "    # Permanent update\n",
    "    model.update(update_data=update_data)\n",
    "    output_data = model.calculate_state_estimation()\n",
    "    print(\"Success\")\n",
    "except PowerGridError as e:\n",
    "    print(e)\n",
    "finally:\n",
    "    # fully reinstigate the original model to prevent this cell from influencing re-runs. A model update would be sufficient as well.\n",
    "    model = PowerGridModel(input_data)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3da1f2aa",
   "metadata": {},
   "source": [
    "## Batch calculation\n",
    "\n",
    "For state estimation a batch calculation can be performed in a similar manner as for powerflow calculations. Below is a short example provided for reference. For more a more detailed example about performing a batch calculation, please refer to the [Power Flow Example](Power%20Flow%20Example.ipynb) or visit the [Documentation](../user_manual/calculations.md)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c90cc3f8",
   "metadata": {},
   "source": [
    "### Create Update Dataset\n",
    "The following code creates an update dataset which runs state estimation for a batch with three different measurements for the symmetric voltage sensors\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "303af910",
   "metadata": {},
   "outputs": [],
   "source": [
    "sym_voltage_sensor_update = initialize_array(\n",
    "    DatasetType.update, ComponentType.sym_voltage_sensor, (4, 3)\n",
    ")  # 4 scenarios, 3 objects per scenario\n",
    "# for each mutation, only one object is specified\n",
    "sym_voltage_sensor_update[\"id\"] = [[11, 12, 13]] * 4\n",
    "sym_voltage_sensor_update[\"u_measured\"] = [\n",
    "    [10402.31, 10023.68, 10095.40],\n",
    "    [10583.20, 9869.85, 10105.28],\n",
    "    [10482.87, 9850.85, 10140.77],\n",
    "    [10482.87, 9850.85, 10140.77],\n",
    "]\n",
    "sym_voltage_sensor_update[\"u_sigma\"][2, 2] = np.inf  # disable the third sensor of the third scenario\n",
    "\n",
    "sym_power_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_power_sensor, (4, 1))\n",
    "sym_power_sensor_update[\"id\"] = [21]\n",
    "sym_power_sensor_update[\"power_sigma\"] = [\n",
    "    [1.0e3],\n",
    "    [1.0e3],\n",
    "    [1.0e3],\n",
    "    [np.inf],  # disables this sensor on the last scenario\n",
    "]\n",
    "\n",
    "update_data = {\n",
    "    ComponentType.sym_voltage_sensor: sym_voltage_sensor_update,\n",
    "    ComponentType.sym_power_sensor: sym_power_sensor_update,\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bbf1d832",
   "metadata": {},
   "source": [
    "### Validation (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "70f761a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model.validation import assert_valid_batch_data\n",
    "\n",
    "assert_valid_batch_data(\n",
    "    input_data=input_data, update_data=update_data, calculation_type=CalculationType.state_estimation\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "68d935f0",
   "metadata": {},
   "source": [
    "### Call Update Method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4d9d5de2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[10466.15445067  9974.61085414 10078.90372425]\n",
      " [10480.594761    9987.299152   10094.19520541]\n",
      " [10416.3953498   9920.53736882 10027.28386252]\n",
      " [10452.49360638  9957.85948757 10066.06548575]]\n"
     ]
    }
   ],
   "source": [
    "batch_output_data = model.calculate_state_estimation(\n",
    "    update_data=update_data,\n",
    "    symmetric=True,\n",
    "    error_tolerance=1e-8,\n",
    "    max_iterations=20,\n",
    "    calculation_method=CalculationMethod.iterative_linear,\n",
    ")\n",
    "print(batch_output_data[ComponentType.node][\"u\"])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}